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Amendments to the Claims: 

This listing of claims will replace all prior versions, and listings of claims in the application: 
Listing of Claims: 

1 1 . (Currently amended) A method of displaying multimedia information 

2 stored in a multimedia document on a display, the multimedia information comprising 

3 information of a plurality of types including information of a first type and information of a 

4 second type, the method comprising: 

5 displaying a graphical user interface (GUI) on the display; 

6 displaying, in a first area of the GUI, a representation of the multimedia 

7 information stored by the multimedia document, the displayed representation of the multimedia 

8 information comprising a representation of information of the first type and a representation of 

9 information of the second type; 

10 displaying a first lens moveable in response to user input over representations of 

1 1 multimedia information displayed in the GUI, the first lens covering a first portion of the first 

12 area; and 

13 displaying, in a second area of the GUI, a representation of multimedia 



14 information displayed in the first portion of the first area, the representation of multimedia 

15 information displayed in the second area comprising a portion of the representation of 

16 information of the first type covered by the first lens and a portion of the representation of 

1 7 information of the second type covered by the first lens. 



1 2. (Original) The method of claim 1 wherein displaying the representation of 

2 the multimedia information stored by the multimedia document in the first area of the GUI 

3 comprises: 

4 displaying a first thumbnail image in the first area of the GUI, the first thumbnail 

5 image comprising the representation of information of the first type; and 
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6 displaying a second thumbnail image in the first area of the GUI, the second 

7 thumbnail image comprising the representation of information of the second type. 

1 3. (Original) The method of claim 1 wherein displaying, in the second area 

2 of the GUI, the representation of multimedia information displayed in the first portion of the first 

3 area comprises: 

4 displaying the portion of the representation of information of the first type 

5 covered by the first lens in a first panel in the second area of the GUI; and 

6 displaying the portion of the representation of information of the second type 

7 covered by the first lens in a second panel in the second area of the GUI. 

1 4. (Original) The method of claim 1 wherein displaying, in the second area 

2 of the GUI, the representation of multimedia information displayed in the first portion of the first 

3 area comprises: 

4 determining a first time and a second time associated with the first lens; 

5 displaying, in the second area of the GUI, a representation of information of the 

6 first type occurring between the first time and the second time associated with the first lens; and 

7 displaying, in the second area of the GUI, a representation of information of the 

8 second type occurring between the first time and the second time associated with the first lens. 

1 5. (Original) The method of claim 1 further comprising: 

2 receiving user input moving the first lens to cover a second portion of the first 

3 area; and 

4 responsive to the user input, automatically changing the information displayed in 

5 the second area of the GUI such that the representation of multimedia information displayed in 

6 the second area of the GUI corresponds to the representation of multimedia information included 

7 in the second portion of the first area. 

1 6. (Currently amended) The method of claim 1 further comprising: 
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2 displaying a second lens moveable in response to user input over representations 

3 of multimedia information displayed in the GUI, the second lens covering a first portion of the 

4 second area; and 

5 displaying, in a third area of the GUI, a representation of multimedia information 



6 corresponding to the first portion of the second area, the representation of multimedia 

7 information displayed in the third area comprising a portion of the representation of information 

8 of the first type covered by the second lens and a portion of the representation of information of 

9 the second type covered by the second lens. 



1 7. (Original) The method of claim 6 wherein displaying, in the third area of 

2 the GUI, the representation of multimedia information corresponding to the first portion of the 

3 second area comprises: 

4 determining a first time and a second time associated with the second lens; 

5 displaying, in the third area of the GUI, a representation of information of the first 

6 type occurring between the first time and the second time associated with the second lens; and 

7 displaying, in the third area of the GUI, a representation of information of the 

8 second type occurring between the first time and the second time associated with the second lens. 

1 8. (Original) The method of claim 6 wherein: 

2 displaying the representation of the multimedia information stored by the 

3 multimedia document in the first area of the GUI comprises: 

4 displaying a first thumbnail image in the first area of the GUI, the first 

5 thumbnail image comprising the representation of information of the first type; and 

6 displaying a second thumbnail image in the first area of the GUI, the 

7 second thumbnail image comprising the representation of information of the second type; 

8 displaying the representation of multimedia information displayed in the first 

9 portion of the first area in the second area of the GUI comprises: 

1 0 displaying the portion of the representation of information of the first type 

1 1 covered by the first lens in a first panel in the second area of the GUI; and 
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12 displaying the portion of the representation of information of the second 

13 type covered by the first lens in a second panel in the second area of the GUI; and 

14 displaying the representation of multimedia information corresponding to the first 

15 portion of the second area in the third area of the GUI comprises: 

16 displaying the representation of information of the first type corresponding 

17 to the first portion of the second area of the GUI in a first sub-area of the third area of the GUI; 

18 and 

19 displaying the representation of information of the second type 

20 corresponding to the first portion of the second area of the GUI in a second sub-area of the third 

21 area of the GUI. 

1 9. (Original) The method of claim 6 further comprising: 

2 receiving a user input moving the second lens to cover a second portion of the 

3 second area; and 

4 responsive to the user input, automatically changing the information displayed in 

5 the third area of the GUI such that the representation of multimedia information displayed in the 

6 third area of the GUI corresponds to the representation of the multimedia information included in 

7 the second portion of the second area. 

1 10. (Original) The method of claim 6 further comprising: 

2 receiving a user input moving the first lens to cover a second portion of the first 

3 area; and 

4 responsive to the user input, automatically: 

5 changing the information displayed in the second area of the GUI such 

6 that the representation of multimedia information displayed in the second area of the GUI 

7 corresponds to the representation of multimedia information included in the second portion of 

8 the first area; and 

9 changing the information displayed in the third area of the GUI such that 
10 the representation of multimedia information displayed in the third area of the GUI corresponds 
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1 1 to the representation of the multimedia information included in the second portion of the second 

12 area. 

1 11. (Original) The method of claim 6 further comprising: 

2 displaying a sub-lens covering a portion of the first area of the GUI corresponding 

3 to the first portion of the second area of the GUI covered by the second lens. 

1 12. (Original) The method of claim 1 1 further comprising: 

2 receiving a user input moving the second lens to cover a second portion of the 

3 second area; and 

4 responsive to the user input, automatically changing a position of the sub-lens to 

5 cover a portion of the first area of the GUI corresponding to the second portion of the second 

6 area. 

1 13. (Original) The method of claim 1 wherein: 

2 the information of the first type corresponds to video information; and 

3 the representation of the information of the first type comprises one or more video 

4 keyframes extracted from the video information. 

1 14. (Original) The method of claim 13 wherein: 

2 the information of the second type corresponds to audio information; and 

3 the representation of information of the second type comprises text information 

4 obtained from transcribing the audio information. 

1 15. (Original) The method of claim 13 wherein: 

2 the information of the second type corresponds to closed-caption (CC) text 

3 information; and 

4 the representation of information of the second type comprises text information 

5 included in the CC text information. 

1 16. (Original) The method of claim 1 further comprising: 
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2 receiving information indicating a user-specified concept of interest; and 

3 analyzing the multimedia information stored in the multimedia document to 

4 identify one or more locations in the multimedia information that are relevant to the user- 

5 specified concept of interest; 

6 wherein displaying the representation of multimedia information in the first area 

7 of the GUI comprises annotating the one or more locations in the multimedia information that 

8 are relevant to the user-specified concept of interest; and 

9 wherein displaying, in the second area of the GUI, a representation of multimedia 

10 information displayed in the first portion of the first area comprises annotating the one or more 

1 1 locations in the multimedia information that are relevant to the user-specified concept of interest 

12 and that are located in the first portion of the first area. 

1 17. (Original) The method of claim 1 further comprising: 

2 receiving input indicating selection of a portion of the multimedia information 

3 occurring between a first time and a second time; and 

4 performing a first operation on the portion of the multimedia information 

5 occurring between a first time and a second time. 

1 18. (Currently amended) A method of displaying multimedia information 

2 stored in a multimedia document on a display, the multimedia information comprising 

3 information of a first type and information of a second type, the method comprising: 

4 displaying a graphical user interface (GUI) on the display; 

5 displaying, in a first area of the GUI, a representation of the multimedia 

6 information stored by the multimedia document occurring between a start time (t s ) and an end 

7 time (t e ) associated with the multimedia document, the displayed representation of the 

8 multimedia information comprising a representation of information of the first type occurring 

9 between t s and t e and a representation of information of the second type occurring between t s 
10 and t e , where (t e > t s ); 
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1 1 displaying a first lens moveable in response to user input over representations of 

12 multimedia information displayed in the GUI, the first lens emphasizing a portion of the first 

13 area of the GUI, the portion of the first area emphasized by the first lens comprising a 

14 representation of multimedia information occurring between a first time (ti ) and a second time 

1 5 (t2), where (t s < ti < t2 < t e ); and 

16 displaying, in a second area of the GUI, the representation of multimedia 

17 information occurring between t^ and t2, the representation of multimedia information displayed 

1 8 in the second area comprising a representation of information of the first type occurring between 

19 t\ and t2 and a representation of information of the second type occurring between t\ and t2- 

1 19. (Currently amended) The method of claim 18 further comprising: 

2 displaying a second lens moveable in response to user input over representations 

3 of multimedia information displayed in the GUI, the second lens emphasizing a portion of the 

4 second area of the GUI, the portion of the second area emphasized by the second lens comprising 

5 a representation of multimedia information occurring between a third time (t3) and a fourth time 

6 (t4), where (tj < t3 < t4 < t2>; and 

7 displaying, in a third area of the GUI, the representation of multimedia 

8 information occurring between t3 and t4, the representation of multimedia information displayed 

9 in the third area comprising a representation of information of the first type occurring between t3 
10 and t4 and a representation of information of the second type occurring between t3 and t4. 

1 20. (Original) The method of claim 19 further comprising: 

2 changing the position of the first lens in response to user input such that the first 

3 lens emphasizes a portion of the first area of the GUI comprising a representation of multimedia 

4 information occurring between a fifth time (t5) and a sixth time (tg), where (t s < 15 < tg < t e ), 

5 (t 5 * tj ), and (tg 5t t2); and 

6 responsive to the change in the position of the first lens, automatically displaying, 

7 in the second area of the GUI, the representation of multimedia information occurring between t5 
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8 and tg, the representation of multimedia information displayed in the second area comprising a 

9 representation of information of the first type occurring between 15 and tg and a representation of 
10 information of the second type occurring between t5 and tg. 



1 21. (Original) The method of claim 19 further comprising: 

2 changing the position of the second lens in response to user input such that the 

3 second lens emphasizes a portion of the second area of the GUI comprising a representation of 

4 multimedia information occurring between a fifth time (t5) and a sixth time (tg), where (t\ <ts< 

5 tg < t2), (t5 * t3), and (tg * t4>; and 

6 responsive to the change in the position of the second lens, automatically 



7 displaying, in the third area of the GUI, the representation of multimedia information occurring 

8 between t5 and tg, the representation of multimedia information displayed in the third area 

9 comprising a representation of information of the first type occurring between 15 and tg and a 
10 representation of information of the second type occurring between t5 and tg. 



1 22. (Original) The method of claim 19 further comprising: 

2 displaying a third lens emphasizing a portion of the first area of the GUI 

3 comprising a representation of multimedia information occurring between t3 and 14 

1 23. (Original) The method of claim 22 further comprising: 

2 changing the position of the second lens in response to user input such that the 

3 second lens emphasizes a portion of the second area of the GUI comprising a representation of 

4 multimedia information occurring between a fifth time (15) and a sixth time (tg), where (t \ <t$< 

5 tg<t2),(t 5 *t 3 ),and(tg*t4);and 

6 responsive to the change in the position of the second lens, automatically 

7 changing the position of the third lens such that the third lens emphasizes a portion of the first 

8 area of the GUI comprising a representation of multimedia information occurring between t5 and 

9 tg. 
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1 24. (Original) The method of claim 1 8 wherein: 

2 the information of the first type is video information; 

3 the information of the second type is audio information; 

4 the representation of the information of the first type comprises one or more video 

5 keyframes extracted from the video information; and 

6 the representation of information of the second type comprises text information 

7 obtained from transcribing the audio information. 

1 25. (Original) The method of claim 18 wherein: 

2 the information of the first type is video information; 

3 the information of the second type is closed-caption (CC) text information; 

4 the representation of the information of the first type comprises one or more video 

5 keyframes extracted from the video information; and 

6 the representation of the information of the second type comprises text 

7 information included in the CC text information. 

1 26. (Original) The method of claim 18 further comprising: 

2 receiving information indicating a first topic; and 

3 analyzing the multimedia information stored in the multimedia document to 

4 identify one or more locations in the multimedia information that are relevant to the first topic; 

5 wherein displaying the representation of the multimedia information stored by the 

6 multimedia document occurring between t s and t e in the first area of the GUI comprises 

7 highlighting the one or more locations in the multimedia information displayed in the first area 

8 of the GUI; and 

9 wherein displaying the representation of multimedia information occurring 

10 between t\ and t2 in the second area of the GUI comprises highlighting the one or more locations 

1 1 in the multimedia information that occur between times t\ and t2- 

1 27. (Original) The method of claim 18 further comprising: 
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2 receiving input indicating selection of a portion of the multimedia information 

3 occurring between a selection start time and a selection end time; and 

4 performing a first operation on the portion of the multimedia information 

5 occurring between the selection start time and the selection end time. 

1 28. (Currently amended) A method of displaying multimedia information 

2 stored in a multimedia document on a display, the multimedia information comprising video 

3 information and information of a first type, the method comprising: 

4 displaying a graphical user interface (GUI) on the display; 

5 displaying, in a first section of a first area of the GUI, a first set of one or more 

6 video keyframes extracted from the video information occurring between a start time (t s ) and an 

7 end time (t e ) associated with the multimedia document, where (t e > t s ); 

8 displaying, in a second section of the first area of the GUI, text information 

9 corresponding to the information of the first type occurring between t s and t e ; 

10 displaying a first lens moveable in response to user input over representations of 



1 1 multimedia information displayed in the GUI, the first lens emphasizing a portion of the first 

12 section of the first area occurring between a first time (ti ) and a second time (12) and a portion of 

13 the second section of the first area occurring between ti and t2, the emphasized portion of the 

14 first section of the first area comprising a second set of one or more video keyframes extracted 

15 from the video information occurring between t\ and t2, the emphasized portion of the second 

16 section of the first area comprising text information corresponding to information of the first type 

17 occurring between t\ and t2, wherein the second set of one or more keyframes is a subset of the 

1 8 first set of one or more keyframes and (t s < t\ < t2 < t e ); 



19 displaying the second set of one or more keyframes in a first section of a second 

20 area of the GUI; and 

21 displaying text information corresponding to the information of the first type 

22 occurring between t \ and t2 in a second section of the second area of the GUI. 
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1 29. (Currently amended) The method of claim 28 further comprising: 

2 displaying a second lens moveable in response to user input over representations 

3 of multimedia information displayed in the GUI, the second lens emphasizing a portion of the 

4 first section of the second area and a portion of the second section of the second area, the 

5 emphasized portion of the first section of the second area comprising a third set of one or more 

6 video keyframes extracted from the video information occurring between a third time O3) and a 

7 fourth time (14), the emphasized portion of the second section of the second area comprising text 

8 information corresponding to information of the first type occurring between t3 and 14, wherein 

9 the third set of one or more video keyframes is a subset of the second set of one or more video 
10 keyframes and (tj < 13 < 14 < t2); 



1 1 displaying a keyframe from the third set of one or more keyframes in a first 

12 section of a third area of the GUI; and 

13 displaying text information corresponding to the information of the first type 

14 occurring between t3 and 14 in a second section of the third area of the GUI. 

1 30. (Original) The method of claim 28 further comprising: 

2 displaying a second lens emphasizing a portion of the first section of the second 



3 area and a portion of the second section of the second area, the emphasized portion of the first 

4 section of the second area comprising a third set of one or more video keyframes extracted from 

5 the video information occurring between a third time (t3) and a fourth time (ty), the emphasized 

6 portion of the second section of the second area comprising text information corresponding to 

7 information of the first type occurring between t3 and t4, wherein the third set of one or more 

8 video keyframes is a subset of the second set of one or more video keyframes and (t \ < 13 < 14 < 

9 t 2 ); 

10 outputting video information starting from t3 or from t4 or from a time between t3 

1 1 and 14 in a first section of a third area of the GUI; and 
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12 displaying text information corresponding to the information of the first type 

13 occurring between t3 and 14 in a second section of the third area of the GUI. 

1 31. (Original) The method of claim 28 wherein the information of the first 

2 type is audio information, and the text information corresponding to the information of the first 

3 type is obtained from transcribing the audio information. 

1 32. (Original) The method of claim 28 wherein the information of the first 

2 type is closed-caption (CC) text information, and the text information corresponding to the 

3 information of the first type is extracted from the CC text information. 

1 33. (Original) The method of claim 28 wherein the multimedia information 

2 stored by the multimedia document further comprises slides information, the method comprising: 

3 displaying, in a third section of the first area of the GUI, a first set of one or more 

4 slides extracted from the slides information occurring between t s and t e , wherein the first lens 

5 emphasizes a portion of the third section of the first area comprising a second set of one or more 

6 slides extracted from the slides information occurring between t\ and t2, the second set of one or 

7 more slides is a subset of the first set of one or more slides; and 

8 displaying the second set of one or more slides in a third section of the second 

9 area of the GUI. 

1 34. (Original) The method of claim 33 further comprising: 

2 displaying a second lens emphasizing a portion of the first section of the second 

3 area, a portion of the second section of the second area, and a portion of the third section of the 

4 second area, the emphasized portion of the first section of the second area comprising a third set 

5 of one or more video keyframes extracted from the video information occurring between a third 

6 time (13) and a fourth time (14), the emphasized portion of the second section of the second area 

7 comprising text information corresponding to information of the first type occurring between t3 

8 and 14, the emphasized portion of the third section of the second area comprising a third set of 



Page 13 of 52 



Appl. No. 10/081,129 PATENT 

Amdt. dated February 11, 2008 

Reply to Office Action of April 20, 2007 

9 one or more slides extracted from the slides information occurring between 13 and t4, wherein 

10 the third set of one or more video keyframes is a subset of the second set of one or more video 

1 1 keyframes, the third set of one or more slides is a subset of the second set of one or more slides, 

12 and (tj < t 3 < t 4 < t 2 ); 



13 displaying at least one keyframe from the third set of one or more video 

14 keyframes in a first section of a third area of the GUI; 

15 displaying text information corresponding to the information of the first type 

16 occurring between t 3 and 14 in a second section of the third area of the GUI; and 

17 displaying at least one slide from the third set of one or more slides in a third 

18 section of the third area of the GUI. 

1 35. (Original) The method of claim 28 wherein the multimedia information 

2 stored by the multimedia document further comprises whiteboard images information, the 

3 method comprising: 

4 displaying, in a third section of the first area of the GUI, a first set of one or more 

5 whiteboard images extracted from the whiteboard images information occurring between t s and 



6 t e , wherein the first lens emphasizes a portion of the third section of the first area comprising a 

7 second set of one or more whiteboard images extracted from the whiteboard images information 

8 occurring between t\ and t 2 , the second set of one or more whiteboard images is a subset of the 

9 first set of one or more whiteboard images; and 



10 displaying the second set of one or more whiteboard images in a third section of 

1 1 the second area of the GUI. 

1 36. (Original) The method of claim 35 further comprising: 

2 displaying a second lens emphasizing a portion of the first section of the second 

3 area, a portion of the second section of the second area, and a portion of the third section of the 

4 second area, the emphasized portion of the first section of the second area comprising a third set 

5 of one or more video keyframes extracted from the video information occurring between a third 
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6 time (13) and a fourth time (14), the emphasized portion of the second section of the second area 

7 comprising text information corresponding to information of the first type occurring between t3 

8 and ty, the emphasized portion of the third section of the second area comprising a third set of 

9 one or more whiteboard images extracted from the whiteboard images information occurring 

10 between t3 and 14, wherein the third set of one or more video keyframes is a subset of the second 

1 1 set of one or more video keyframes, the third set of one or more whiteboard images is a subset of 

12 the second set of one or more whiteboard images, and (t\ < t3 < t4 < t2); 



13 displaying at least one keyframe from the third set of one or more video 

14 keyframes in a first section of a third area of the GUI; 

15 displaying text information corresponding to the information of the first type 

16 occurring between t3 and 14 in a second section of the third area of the GUI; and 

17 displaying a whiteboard images from the third set of one or more whiteboard 

1 8 images in a third section of the third area of the GUI. 

1 37. (Currently amended) A system for displaying multimedia information 

2 stored in a multimedia document on a display, the multimedia information comprising 

3 information of a plurality of types including information of a first type and information of a 

4 second type, the system comprising: 

5 means for displaying a graphical user interface (GUI) on the display; 

6 means for displaying, in a first area of the GUI, a representation of the multimedia 

7 information stored by the multimedia document, the displayed representation of the multimedia 

8 information comprising a representation of information of the first type and a representation of 

9 information of the second type; 

10 means for displaying a first lens moveable in response to user input over 

1 1 representations of multimedia information displayed in the GUI, the first lens covering a first 

12 portion of the first area; and 

13 means for displaying, in a second area of the GUI, a representation of multimedia 

14 information displayed in the first portion of the first area, the representation of multimedia 
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15 information displayed in the second area comprising a portion of the representation of 

16 information of the first type covered by the first lens and a portion of the representation of 

1 7 information of the second type covered by the first lens. 

1 38. (Currently amended) A system for displaying multimedia information 

2 stored in a multimedia document on a display, the multimedia information comprising 

3 information of a first type and information of a second type, the system comprising: 

4 means for displaying a graphical user interface (GUI) on the display; 

5 means for displaying, in a first area of the GUI, a representation of the multimedia 



6 information stored by the multimedia document occurring between a start time (t s ) and an end 

7 time (t e ) associated with the multimedia document, the displayed representation of the 

8 multimedia information comprising a representation of information of the first type occurring 

9 between t s and t e and a representation of information of the second type occurring between t s 

1 0 and t e , where (t e > t s ); 

1 1 means for displaying a first lens moveable in response to user input over 

12 representations of multimedia information displayed in the GUI, the first lens emphasizing a 

13 portion of the first area of the GUI, the portion of the first area emphasized by the first lens 

14 comprising a representation of multimedia information occurring between a first time (ti ) and a 

15 second time O2), where (t s < ti < t2 < t e ); and 

16 means for displaying, in a second area of the GUI, the representation of 

17 multimedia information occurring between t^ and t2, the representation of multimedia 

1 8 information displayed in the second area comprising a representation of information of the first 

19 type occurring between t\ and t2 and a representation of information of the second type 

20 occurring between t\ and t2- 

1 39. (Currently amended) A system for of displaying multimedia information 

2 stored in a multimedia document on a display, the multimedia information comprising video 

3 information and information of a first type, the system comprising: 
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4 means for displaying a graphical user interface (GUI) on the display; 

5 means for displaying, in a first section of a first area of the GUI, a first set of one 

6 or more video keyframes extracted from the video information occurring between a start time (t s ) 

7 and an end time (t e ) associated with the multimedia document, where (t e > t s ); 

8 means for displaying, in a second section of the first area of the GUI, text 

9 information corresponding to the information of the first type occurring between t s and t e ; 
10 means for displaying a first lens moveable in response to user input over 



1 1 representations of multimedia information displayed in the GUI, the first lens emphasizing a 

12 portion of the first section of the first area occurring between a first time (ti ) and a second time 

13 02) and a portion of the second section of the first area occurring between t\ and t 2 , the 

14 emphasized portion of the first section of the first area comprising a second set of one or more 

15 video keyframes extracted from the video information occurring between t\ and t 2 , the 

1 6 emphasized portion of the second section of the first area comprising text information 

17 corresponding to information of the first type occurring between t \ and t 2 , wherein the second 

1 8 set of one or more keyframes is a subset of the first set of one or more keyframes and (t s < ti < 

19 t 2 <t e ); 



20 means for displaying the second set of one or more keyframes in a first section of 

21 a second area of the GUI; and 

22 means for displaying text information corresponding to the information of the first 

23 type occurring between t\ and t 2 in a second section of the second area of the GUI. 

1 40. (Currently amended) A computer program product stored on a computer- 

2 readable storage medium for displaying multimedia information stored in a multimedia 

3 document on a display, the multimedia information comprising information of a plurality of 

4 types including information of a first type and information of a second type, the computer 

5 program product comprising: 

6 code for displaying a graphical user interface (GUI) on the display; 
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7 code for displaying, in a first area of the GUI, a representation of the multimedia 

8 information stored by the multimedia document, the displayed representation of the multimedia 

9 information comprising a representation of information of the first type and a representation of 

1 0 information of the second type; 

1 1 code for displaying a first lens moveable in response to user input over 

12 representations of multimedia information displayed in the GUI covering a first portion of the 

13 first area; and 

14 code for displaying, in a second area of the GUI, a representation of multimedia 

15 information displayed in the first portion of the first area, the representation of multimedia 

16 information displayed in the second area comprising a portion of the representation of 

17 information of the first type covered by the first lens and a portion of the representation of 

1 8 information of the second type covered by the first lens. 

1 41 . (Original) The computer program product of claim 40 wherein the code 

2 for displaying the representation of the multimedia information stored by the multimedia 

3 document in the first area of the GUI comprises: 

4 code for displaying a first thumbnail image in the first area of the GUI, the first 

5 thumbnail image comprising the representation of information of the first type; and 

6 code for displaying a second thumbnail image in the first area of the GUI, the 

7 second thumbnail image comprising the representation of information of the second type. 

1 42. (Original) The computer program product of claim 40 wherein the code 

2 for displaying, in the second area of the GUI, the representation of multimedia information 

3 displayed in the first portion of the first area comprises: 

4 code for displaying the portion of the representation of information of the first 

5 type covered by the first lens in a first panel in the second area of the GUI; and 

6 code for displaying the portion of the representation of information of the second 

7 type covered by the first lens in a second panel in the second area of the GUI. 
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1 43. (Original) The computer program product of claim 40 wherein the code 

2 for displaying, in the second area of the GUI, the representation of multimedia information 

3 displayed in the first portion of the first area comprises: 

4 code for determining a first time and a second time associated with the first lens; 

5 code for displaying, in the second area of the GUI, a representation of information 

6 of the first type occurring between the first time and the second time associated with the first 

7 lens; and 

8 code for displaying, in the second area of the GUI, a representation of information 

9 of the second type occurring between the first time and the second time associated with the first 
10 lens. 

1 44. (Original) The computer program product of claim 40 further comprising: 

2 code for receiving user input moving the first lens to cover a second portion of the 

3 first area; and 

4 code for responsive to the user input, automatically changing the information 

5 displayed in the second area of the GUI such that the representation of multimedia information 

6 displayed in the second area of the GUI corresponds to the representation of multimedia 

7 information included in the second portion of the first area. 

1 45. (Currently amended) The computer program product of claim 40 further 

2 comprising: 

3 code for displaying a second lens moveable in response to user input over 

4 representations of multimedia information displayed in the GUI, the second lens covering a first 

5 portion of the second area; and 

6 code for displaying, in a third area of the GUI, a representation of multimedia 

7 information corresponding to the first portion of the second area, the representation of 

8 multimedia information displayed in the third area comprising a portion of the representation of 

9 information of the first type covered by the second lens and a portion of the representation of 
1 0 information of the second type covered by the second lens. 
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1 46. (Original) The computer program product of claim 45 wherein the code 

2 for displaying, in the third area of the GUI, the representation of multimedia information 

3 corresponding to the first portion of the second area comprises: 

4 code for determining a first time and a second time associated with the second 

5 lens; 

6 code for displaying, in the third area of the GUI, a representation of information 

7 of the first type occurring between the first time and the second time associated with the second 

8 lens; and 

9 code for displaying, in the third area of the GUI, a representation of information 

10 of the second type occurring between the first time and the second time associated with the 

1 1 second lens. 

1 47. (Original) The computer program product of claim 45 wherein: 

2 the code for displaying the representation of the multimedia information stored by 

3 the multimedia document in the first area of the GUI comprises: 

4 code for displaying a first thumbnail image in the first area of the GUI, the 

5 first thumbnail image comprising the representation of information of the first type; and 

6 code for displaying a second thumbnail image in the first area of the GUI, 

7 the second thumbnail image comprising the representation of information of the second type; 

8 the code for displaying the representation of multimedia information displayed in 

9 the first portion of the first area in the second area of the GUI comprises: 

10 code for displaying the portion of the representation of information of the 

1 1 first type covered by the first lens in a first panel in the second area of the GUI; and 

12 code for displaying the portion of the representation of information of the 

13 second type covered by the first lens in a second panel in the second area of the GUI; and 

14 the code for displaying the representation of multimedia information 

15 corresponding to the first portion of the second area in the third area of the GUI comprises: 
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1 6 code for displaying the representation of information of the first type 

17 corresponding to the first portion of the second area of the GUI in a first sub-area of the third 

18 area of the GUI; and 

19 code for displaying the representation of information of the second type 

20 corresponding to the first portion of the second area of the GUI in a second sub-area of the third 

21 area of the GUI. 

1 48. (Original) The computer program product of claim 45 further comprising: 

2 code for receiving a user input moving the second lens to cover a second portion 

3 of the second area; and 

4 responsive to the user input, code for automatically changing the information 

5 displayed in the third area of the GUI such that the representation of multimedia information 

6 displayed in the third area of the GUI corresponds to the representation of the multimedia 

7 information included in the second portion of the second area. 

1 49. (Original) The computer program product of claim 45 further comprising: 

2 code for receiving a user input moving the first lens to cover a second portion of 

3 the first area; and 

4 responsive to the user input, code for automatically: 

5 changing the information displayed in the second area of the GUI such 

6 that the representation of multimedia information displayed in the second area of the GUI 

7 corresponds to the representation of multimedia information included in the second portion of 

8 the first area; and 

9 changing the information displayed in the third area of the GUI such that 

10 the representation of multimedia information displayed in the third area of the GUI corresponds 

11 to the representation of the multimedia information included in the second portion of the second 

12 area. 

1 50. (Original) The computer program product of claim 45 further comprising: 
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2 code for displaying a sub-lens covering a portion of the first area of the GUI 

3 corresponding to the first portion of the second area of the GUI covered by the second lens. 

1 51. (Original) The computer program product of claim 50 further comprising: 

2 code for receiving a user input moving the second lens to cover a second portion 

3 of the second area; and 

4 responsive to the user input, code for automatically changing a position of the 

5 sub-lens to cover a portion of the first area of the GUI corresponding to the second portion of the 

6 second area. 

1 52. (Original) The computer program product of claim 40 wherein: 

2 the information of the first type corresponds to video information; and 

3 the representation of the information of the first type comprises one or more video 

4 keyframes extracted from the video information. 

1 53. (Original) The computer program product of claim 52 wherein: 

2 the information of the second type corresponds to audio information; and 

3 the representation of information of the second type comprises text information 

4 obtained from transcribing the audio information. 

1 54. (Original) The computer program product of claim 52 wherein: 

2 the information of the second type corresponds to closed-caption (CC) text 

3 information; and 

4 the representation of information of the second type comprises text information 

5 included in the CC text information. 

1 55. (Original) The computer program product of claim 40 further comprising: 

2 code for receiving information indicating a user-specified concept of interest; and 

3 code for analyzing the multimedia information stored in the multimedia document 

4 to identify one or more locations in the multimedia information that are relevant to the user- 

5 specified concept of interest; 
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6 wherein the code for displaying the representation of multimedia information in 

7 the first area of the GUI comprises code for annotating the one or more locations in the 

8 multimedia information that are relevant to the user-specified concept of interest; and 

9 wherein the code for displaying, in the second area of the GUI, a representation of 

10 multimedia information displayed in the first portion of the first area comprises code for 

1 1 annotating the one or more locations in the multimedia information that are relevant to the user- 

12 specified concept of interest and that are located in the first portion of the first area. 

1 56. (Original) The computer program product of claim 40 further comprising: 

2 code for receiving input indicating selection of a portion of the multimedia 

3 information occurring between a first time and a second time; and 

4 code for performing a first operation on the portion of the multimedia information 

5 occurring between a first time and a second time. 

1 57. (Currently amended) A computer program product stored on a computer- 

2 readable storage medium for displaying multimedia information stored in a multimedia 

3 document on a display, the multimedia information comprising information of a first type and 

4 information of a second type, the computer program product comprising: 

5 code for displaying a graphical user interface (GUI) on the display; 

6 code for displaying, in a first area of the GUI, a representation of the multimedia 



7 information stored by the multimedia document occurring between a start time (t s ) and an end 

8 time (t e ) associated with the multimedia document, the displayed representation of the 

9 multimedia information comprising a representation of information of the first type occurring 

1 0 between t s and t e and a representation of information of the second type occurring between t s 

1 1 and t e , where (t e > t s ); 

12 code for displaying a first lens moveable in response to user input over 

13 representations of multimedia information displayed in the GUI, the first lens emphasizing a 

14 portion of the first area of the GUI, the portion of the first area emphasized by the first lens 
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15 comprising a representation of multimedia information occurring between a first time (ti ) and a 

16 second time (t2), where (t s < < t2 < t e ); and 

17 code for displaying, in a second area of the GUI, the representation of multimedia 

18 information occurring between t\ and t2, the representation of multimedia information displayed 

19 in the second area comprising a representation of information of the first type occurring between 

20 t i and t2 and a representation of information of the second type occurring between t\ and t2- 

1 58. (Currently amended) The computer program product of claim 57 further 

2 comprising: 

3 code for displaying a second lens moveable in response to user input over 

4 representations of multimedia information displayed in the GUI, the second lens emphasizing a 

5 portion of the second area of the GUI, the portion of the second area emphasized by the second 

6 lens comprising a representation of multimedia information occurring between a third time (13) 

7 and a fourth time (14), where (t^ < 13 < 14 < t2); and 

8 code for displaying, in a third area of the GUI, the representation of multimedia 

9 information occurring between t3 and t4, the representation of multimedia information displayed 

1 0 in the third area comprising a representation of information of the first type occurring between t3 

1 1 and 14 and a representation of information of the second type occurring between t3 and t4. 

1 59. (Original) The computer program product of claim 58 further comprising: 

2 code for changing the position of the first lens in response to user input such that 

3 the first lens emphasizes a portion of the first area of the GUI comprising a representation of 

4 multimedia information occurring between a fifth time (15) and a sixth time (tg), where (t s < 15 < 

5 t 6 <t e ),(t 5 ^t 1 ),and(t 6 ^t 2 );and 

6 responsive to the change in the position of the first lens, code for automatically 

7 displaying, in the second area of the GUI, the representation of multimedia information 

8 occurring between t5 and tg, the representation of multimedia information displayed in the 



Page 24 of 52 



Appl. No. 10/081,129 PATENT 



Amdt. dated February 11, 2008 
Reply to Office Action of April 20, 2007 

9 second area comprising a representation of information of the first type occurring between t5 and 
10 tg and a representation of information of the second type occurring between t5 and tg. 

1 60. (Original) The computer program product of claim 58 further comprising: 

2 code for changing the position of the second lens in response to user input such 

3 that the second lens emphasizes a portion of the second area of the GUI comprising a 

4 representation of multimedia information occurring between a fifth time (t5) and a sixth time 

5 (t 6 ), where fa < t 5 < t 6 < t 2 ), (t 5 * t 3 ), and (t 6 * t 4 ); and 

6 code for responsive to the change in the position of the second lens, automatically 



7 displaying, in the third area of the GUI, the representation of multimedia information occurring 

8 between t5 and tg, the representation of multimedia information displayed in the third area 

9 comprising a representation of information of the first type occurring between t5 and tg and a 
10 representation of information of the second type occurring between t5 and tg. 



1 61 . (Original) The computer program product of claim 58 further comprising: 

2 code for displaying a third lens emphasizing a portion of the first area of the GUI 

3 comprising a representation of multimedia information occurring between t 3 and ty_ 

1 62. (Original) The computer program product of claim 61 further comprising: 

2 code for changing the position of the second lens in response to user input such 

3 that the second lens emphasizes a portion of the second area of the GUI comprising a 

4 representation of multimedia information occurring between a fifth time fa) and a sixth time 

5 (tg), where fa < 15 < tg < fa * t3>, and (tg * 14); and 

6 code for responsive to the change in the position of the second lens, automatically 

7 changing the position of the third lens such that the third lens emphasizes a portion of the first 

8 area of the GUI comprising a representation of multimedia information occurring between t5 and 

9 tg. 

1 63. (Original) The computer program product of claim 57 wherein: 
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2 the information of the first type is video information; 

3 the information of the second type is audio information; 

4 the representation of the information of the first type comprises one or more video 

5 keyframes extracted from the video information; and 

6 the representation of information of the second type comprises text information 

7 obtained from transcribing the audio information. 

1 64. (Original) The computer program product of claim 57 wherein: 

2 the information of the first type is video information; 

3 the information of the second type is closed-caption (CC) text information; 

4 the representation of the information of the first type comprises one or more video 

5 keyframes extracted from the video information; and 

6 the representation of the information of the second type comprises text 

7 information included in the CC text information. 

1 65. (Original) The computer program product of claim 57 further comprising: 

2 code for receiving information indicating a first topic; and 

3 code for analyzing the multimedia information stored in the multimedia document 

4 to identify one or more locations in the multimedia information that are relevant to the first topic; 

5 wherein the code for displaying the representation of the multimedia information 

6 stored by the multimedia document occurring between t s and t e in the first area of the GUI 

7 comprises code for highlighting the one or more locations in the multimedia information 

8 displayed in the first area of the GUI; and 

9 wherein the code for displaying the representation of multimedia information 

10 occurring between t\ and t2 in the second area of the GUI comprises code for highlighting the 

1 1 one or more locations in the multimedia information that occur between times t\ and t2- 

1 66. (Original) The computer program product of claim 57 further comprising: 

2 code for receiving input indicating selection of a portion of the multimedia 

3 information occurring between a selection start time and a selection end time; and 
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4 code for performing a first operation on the portion of the multimedia information 

5 occurring between the selection start time and the selection end time. 

1 67. (Currently amended) A computer program product stored on a computer- 

2 readable storage medium for displaying multimedia information stored in a multimedia 

3 document on a display, the multimedia information comprising video information and 

4 information of a first type, the computer program product comprising: 

5 code for displaying a graphical user interface (GUI) on the display; 

6 code for displaying, in a first section of a first area of the GUI, a first set of one or 

7 more video keyframes extracted from the video information occurring between a start time (t s ) 

8 and an end time (t e ) associated with the multimedia document, where (t e > t s ); 

9 code for displaying, in a second section of the first area of the GUI, text 

10 information corresponding to the information of the first type occurring between t s and t e ; 

1 1 code for displaying a first lens moveable in response to user input over 



1 2 representations of multimedia information displayed in the GUI, the first lens emphasizing a 

13 portion of the first section of the first area occurring between a first time (ti ) and a second time 

14 (12) and a portion of the second section of the first area occurring between t \ and t2, the 

15 emphasized portion of the first section of the first area comprising a second set of one or more 

16 video keyframes extracted from the video information occurring between t\ and t2, the 

17 emphasized portion of the second section of the first area comprising text information 

18 corresponding to information of the first type occurring between t\ and t2, wherein the second 

19 set of one or more keyframes is a subset of the first set of one or more keyframes and (t s < t \ < 

20 t 2 < t e ); 



21 code for displaying the second set of one or more keyframes in a first section of a 

22 second area of the GUI; and 

23 code for displaying text information corresponding to the information of the first 

24 type occurring between t \ and t2 in a second section of the second area of the GUI. 
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1 68. (Currently amended) The computer program product of claim 67 further 

2 comprising: 

3 code for displaying a second lens moveable in response to user input over 

4 representations of multimedia information displayed in the GUI, the second lens emphasizing a 

5 portion of the first section of the second area and a portion of the second section of the second 

6 area, the emphasized portion of the first section of the second area comprising a third set of one 

7 or more video keyframes extracted from the video information occurring between a third time 

8 (t3) and a fourth time (14), the emphasized portion of the second section of the second area 

9 comprising text information corresponding to information of the first type occurring between t3 

10 and t4, wherein the third set of one or more video keyframes is a subset of the second set of one 

11 or more video keyframes and (tj < 13 < t4 < t2); 



12 code for displaying a keyframe from the third set of one or more keyframes in a 

13 first section of a third area of the GUI; and 

14 code for displaying text information corresponding to the information of the first 

15 type occurring between t3 and t4 in a second section of the third area of the GUI. 

1 69. (Original) The computer program product of claim 67 further comprising: 

2 code for displaying a second lens emphasizing a portion of the first section of the 



3 second area and a portion of the second section of the second area, the emphasized portion of the 

4 first section of the second area comprising a third set of one or more video keyframes extracted 

5 from the video information occurring between a third time (13) and a fourth time O4), the 

6 emphasized portion of the second section of the second area comprising text information 

7 corresponding to information of the first type occurring between t3 and 14, wherein the third set 

8 of one or more video keyframes is a subset of the second set of one or more video keyframes and 

9 (ti<t 3 <t 4 <t 2 ); 

10 code for outputting video information starting from 13 or from 14 or from a time 

1 1 between t3 and 14 in a first section of a third area of the GUI; and 
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12 code for displaying text information corresponding to the information of the first 

13 type occurring between t3 and 14 in a second section of the third area of the GUI. 

1 70. (Original) The computer program product of claim 67 wherein the 

2 information of the first type is audio information, and the text information corresponding to the 

3 information of the first type is obtained from transcribing the audio information. 

1 71 . (Original) The computer program product of claim 67 wherein the 

2 information of the first type is closed-caption (CC) text information, and the text information 

3 corresponding to the information of the first type is extracted from the CC text information. 

1 72. (Original) The computer program product of claim 67 wherein the 

2 multimedia information stored by the multimedia document further comprises slides information, 

3 the computer program product further comprising: 

4 code for displaying, in a third section of the first area of the GUI, a first set of one 

5 or more slides extracted from the slides information occurring between t s and t e , wherein the 

6 first lens emphasizes a portion of the third section of the first area comprising a second set of one 

7 or more slides extracted from the slides information occurring between t\ and t2, the second set 

8 of one or more slides is a subset of the first set of one or more slides; and 

9 code for displaying the second set of one or more slides in a third section of the 
10 second area of the GUI 

1 73. (Original) The computer program product of claim 72 further comprising: 

2 code for displaying a second lens emphasizing a portion of the first section of the 



3 second area, a portion of the second section of the second area, and a portion of the third section 

4 of the second area, the emphasized portion of the first section of the second area comprising a 

5 third set of one or more video keyframes extracted from the video information occurring between 

6 a third time (t3) and a fourth time (t4), the emphasized portion of the second section of the 

7 second area comprising text information corresponding to information of the first type occurring 

8 between t3 and 14, the emphasized portion of the third section of the second area comprising a 
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9 third set of one or more slides extracted from the slides information occurring between t3 and 14, 

10 wherein the third set of one or more video keyframes is a subset of the second set of one or more 

1 1 video keyframes, the third set of one or more slides is a subset of the second set of one or more 

12 slides, and (tj < t 3 < t 4 < t 2 ); 



13 code for displaying at least one keyframe from the third set of one or more video 

14 keyframes in a first section of a third area of the GUI; 

15 code for displaying text information corresponding to the information of the first 

16 type occurring between t 3 and 14 in a second section of the third area of the GUI; and 

17 displaying at least one slide from the third set of one or more slides in a third 

1 8 section of the third area of the GUI. 

1 74. (Original) The computer program product of claim 67 wherein the 

2 multimedia information stored by the multimedia document further comprises whiteboard 

3 images information, the computer program product further comprising: 

4 code for displaying, in a third section of the first area of the GUI, a first set of one 



5 or more whiteboard images extracted from the whiteboard images information occurring 

6 between t s and t e , wherein the first lens emphasizes a portion of the third section of the first area 

7 comprising a second set of one or more whiteboard images extracted from the whiteboard images 

8 information occurring between t^ and t 2 , the second set of one or more whiteboard images is a 

9 subset of the first set of one or more whiteboard images; and 



10 code for displaying the second set of one or more whiteboard images in a third 

1 1 section of the second area of the GUI. 

1 75. (Original) The computer program product of claim 74 further comprising: 

2 code for displaying a second lens emphasizing a portion of the first section of the 

3 second area, a portion of the second section of the second area, and a portion of the third section 

4 of the second area, the emphasized portion of the first section of the second area comprising a 

5 third set of one or more video keyframes extracted from the video information occurring between 

6 a third time (t 3 ) and a fourth time (ty), the emphasized portion of the second section of the 
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7 second area comprising text information corresponding to information of the first type occurring 

8 between t3 and ty, the emphasized portion of the third section of the second area comprising a 

9 third set of one or more whiteboard images extracted from the whiteboard images information 

10 occurring between t3 and 14, wherein the third set of one or more video keyframes is a subset of 

1 1 the second set of one or more video keyframes, the third set of one or more whiteboard images is 

12 a subset of the second set of one or more whiteboard images, and (t\ < 13 < t4 < 12); 



13 code for displaying at least one keyframe from the third set of one or more video 

14 keyframes in a first section of a third area of the GUI; 

15 code for displaying text information corresponding to the information of the first 

16 type occurring between t3 and t4 in a second section of the third area of the GUI; and 

17 code for displaying a whiteboard images from the third set of one or more 

1 8 whiteboard images in a third section of the third area of the GUI. 

1 76. (Currently amended) A system for displaying multimedia information 

2 stored in a multimedia document, the multimedia information comprising information of a 

3 plurality of types including information of a first type and information of a second type, the 

4 system comprising: 

5 a display; 

6 a processor; and 

7 a memory coupled to the processor, the memory configured to store a plurality of 

8 code modules for execution by the processor, the plurality of code modules comprising: 

9 a code module for displaying a graphical user interface (GUI) on the 

10 display; 

11 a code module for displaying, in a first area of the GUI, a representation of 

12 the multimedia information stored by the multimedia document, the displayed representation of 

13 the multimedia information comprising a representation of information of the first type and a 

14 representation of information of the second type; 
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15 a code module for displaying a first lens moveable in response to user 

16 input over representations of multimedia information displayed in the GUI, the first lens 

17 covering a first portion of the first area; and 

1 8 a code module for displaying, in a second area of the GUI, a 

19 representation of multimedia information displayed in the first portion of the first area, the 

20 representation of multimedia information displayed in the second area comprising a portion of 

21 the representation of information of the first type covered by the first lens and a portion of the 

22 representation of information of the second type covered by the first lens. 

1 77. (Original) The system of claim 76 wherein the code module for 

2 displaying the representation of the multimedia information stored by the multimedia document 

3 in the first area of the GUI comprises: 

4 a code module for displaying a first thumbnail image in the first area of the GUI, 

5 the first thumbnail image comprising the representation of information of the first type; and 

6 a code module for displaying a second thumbnail image in the first area of the 

7 GUI, the second thumbnail image comprising the representation of information of the second 

8 type. 

1 78. (Original) The system of claim 76 wherein the code module for 

2 displaying, in the second area of the GUI, the representation of multimedia information 

3 displayed in the first portion of the first area comprises: 

4 a code module for displaying the portion of the representation of information of 

5 the first type covered by the first lens in a first panel in the second area of the GUI; and 

6 a code module for displaying the portion of the representation of information of 

7 the second type covered by the first lens in a second panel in the second area of the GUI. 

1 79. (Original) The system of claim 76 wherein the code module for 

2 displaying, in the second area of the GUI, the representation of multimedia information 

3 displayed in the first portion of the first area comprises: 
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4 a code module for determining a first time and a second time associated with the 

5 first lens; 

6 a code module for displaying, in the second area of the GUI, a representation of 

7 information of the first type occurring between the first time and the second time associated with 

8 the first lens; and 

9 a code module for displaying, in the second area of the GUI, a representation of 

10 information of the second type occurring between the first time and the second time associated 

1 1 with the first lens. 

1 80. (Original) The system of claim 76 wherein the plurality of code modules 

2 further comprises: 

3 a code module for receiving user input moving the first lens to cover a second 

4 portion of the first area; and 

5 responsive to the user input, a code module for automatically changing the 

6 information displayed in the second area of the GUI such that the representation of multimedia 

7 information displayed in the second area of the GUI corresponds to the representation of 

8 multimedia information included in the second portion of the first area. 

1 81. (Currently amended) The system of claim 76 wherein the plurality of 

2 code modules further comprises: 

3 a code module for displaying a second lens moveable in response to user input 

4 over representations of multimedia information displayed in the GUI, the second lens covering a 

5 first portion of the second area; and 

6 a code module for displaying, in a third area of the GUI, a representation of 

7 multimedia information corresponding to the first portion of the second area, the representation 

8 of multimedia information displayed in the third area comprising a portion of the representation 

9 of information of the first type covered by the second lens and a portion of the representation of 
1 0 information of the second type covered by the second lens. 
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1 82. (Original) The system of claim 81 wherein the code module for 

2 displaying, in the third area of the GUI, the representation of multimedia information 

3 corresponding to the first portion of the second area comprises: 

4 a code module for determining a first time and a second time associated with the 

5 second lens; 

6 a code module for displaying, in the third area of the GUI, a representation of 

7 information of the first type occurring between the first time and the second time associated with 

8 the second lens; and 

9 a code module for displaying, in the third area of the GUI, a representation of 

10 information of the second type occurring between the first time and the second time associated 

1 1 with the second lens. 

1 83. (Original) The system of claim 81 wherein: 

2 the code module for displaying the representation of the multimedia information 

3 stored by the multimedia document in the first area of the GUI comprises: 

4 a code module for displaying a first thumbnail image in the first area of 

5 the GUI, the first thumbnail image comprising the representation of information of the first type; 

6 and 

7 a code module for displaying a second thumbnail image in the first area of 

8 the GUI, the second thumbnail image comprising the representation of information of the second 

9 type; 

10 the code module for displaying the representation of multimedia information 

1 1 displayed in the first portion of the first area in the second area of the GUI comprises: 

12 a code module for displaying the portion of the representation of 

13 information of the first type covered by the first lens in a first panel in the second area of the 

14 GUI; and 

15 a code module for displaying the portion of the representation of 

1 6 information of the second type covered by the first lens in a second panel in the second area of 

17 the GUI; and 
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1 8 the code module for displaying the representation of multimedia information 

19 corresponding to the first portion of the second area in the third area of the GUI comprises: 

20 a code module for displaying the representation of information of the first 

21 type corresponding to the first portion of the second area of the GUI in a first sub-area of the 

22 third area of the GUI; and 

23 a code module for displaying the representation of information of the 

24 second type corresponding to the first portion of the second area of the GUI in a second sub-area 

25 of the third area of the GUI . 

1 84. (Original) The system of claim 8 1 wherein the plurality of code modules 

2 further comprises: 

3 a code module for receiving a user input moving the second lens to cover a 

4 second portion of the second area; and 

5 responsive to the user input, a code module for automatically changing the 

6 information displayed in the third area of the GUI such that the representation of multimedia 

7 information displayed in the third area of the GUI corresponds to the representation of the 

8 multimedia information included in the second portion of the second area. 

1 85. (Original) The system of claim 8 1 wherein the plurality of code modules 

2 further comprises: 

3 a code module for receiving a user input moving the first lens to cover a second 

4 portion of the first area; and 

5 responsive to the user input, a code module for automatically: 

6 changing the information displayed in the second area of the GUI such 

7 that the representation of multimedia information displayed in the second area of the GUI 

8 corresponds to the representation of multimedia information included in the second portion of 

9 the first area; and 

10 changing the information displayed in the third area of the GUI such that 

1 1 the representation of multimedia information displayed in the third area of the GUI corresponds 
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12 to the representation of the multimedia information included in the second portion of the second 

13 area. 

1 86. (Original) The system of claim 8 1 wherein the plurality of code modules 

2 further comprises: 

3 a code module for displaying a sub-lens covering a portion of the first area of the 

4 GUI corresponding to the first portion of the second area of the GUI covered by the second lens. 

1 87. (Original) The system of claim 86 wherein the plurality of code modules 

2 further comprises: 

3 a code module for receiving a user input moving the second lens to cover a 

4 second portion of the second area; and 

5 responsive to the user input, a code module for automatically changing a position 

6 of the sub-lens to cover a portion of the first area of the GUI corresponding to the second portion 

7 of the second area. 

1 88. (Original) The system of claim 76 wherein: 

2 the information of the first type corresponds to video information; and 

3 the representation of the information of the first type comprises one or more video 

4 keyframes extracted from the video information. 

1 89. (Original) The system of claim 88 wherein: 

2 the information of the second type corresponds to audio information; and 

3 the representation of information of the second type comprises text information 

4 obtained from transcribing the audio information. 

1 90. (Original) The system of claim 88 wherein: 

2 the information of the second type corresponds to closed-caption (CC) text 

3 information; and 

4 the representation of information of the second type comprises text information 

5 included in the CC text information. 
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1 91 . (Original) The system of claim 76 wherein the plurality of code modules 

2 further comprises: 

3 a code module for receiving information indicating a user-specified concept of 

4 interest; and 

5 a code module for analyzing the multimedia information stored in the multimedia 

6 document to identify one or more locations in the multimedia information that are relevant to the 

7 user-specified concept of interest; 

8 wherein the code module for displaying the representation of multimedia 

9 information in the first area of the GUI comprises a code module for annotating the one or more 

10 locations in the multimedia information that arc relevant to the user-specified concept of interest; 

11 and 

12 wherein the code module for displaying, in the second area of the GUI, a 

13 representation of multimedia information displayed in the first portion of the first area comprises 

14 a code module for annotating the one or more locations in the multimedia information that are 

15 relevant to the user-specified concept of interest and that are located in the first portion of the 

16 first area. 

1 92. (Original) The system of claim 76 wherein the plurality of code modules 

2 further comprises: 

3 a code module for receiving input indicating selection of a portion of the 

4 multimedia information occurring between a first time and a second time; and 

5 a code module for performing a first operation on the portion of the multimedia 

6 information occurring between a first time and a second time. 

7 93. (Currently amended) A system for displaying multimedia information 

8 stored in a multimedia document, the multimedia information comprising information of a first 

9 type and information of a second type, the system comprising: 

10 a display; 

1 1 a processor; and 
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12 a memory coupled to the processor, the memory configured to store a plurality of 

13 code modules for execution by the processor, the plurality of code modules comprising: 

14 a code module for displaying a graphical user interface (GUI) on the 

15 display; 

16 a code module for displaying, in a first area of the GUI, a representation of 



17 the multimedia information stored by the multimedia document occurring between a start time 

18 (t s ) and an end time (t e ) associated with the multimedia document, the displayed representation 

19 of the multimedia information comprising a representation of information of the first type 

20 occurring between t s and t e and a representation of information of the second type occurring 

2 1 between t s and t e , where (t e > t s ); 

22 a code module for displaying a first lens moveable in response to user 

23 input over representations of multimedia information displayed in the GUI, the first lens 

24 emphasizing a portion of the first area of the GUI, the portion of the first area emphasized by the 

25 first lens comprising a representation of multimedia information occurring between a first time 

26 (ti ) and a second time (t2), where (t s <t\ < t2 < t e ); and 

27 a code module for displaying, in a second area of the GUI, the 

28 representation of multimedia information occurring between ti and t2, the representation of 

29 multimedia information displayed in the second area comprising a representation of information 

30 of the first type occurring between t^ and t2 and a representation of information of the second 

3 1 type occurring between t\ and t2- 



1 94. (Currently amended) The system of claim 93 wherein the plurality of 

2 code modules further comprises: 

3 a code module for displaying a second lens moveable in response to user input 

4 over representations of multimedia information displayed in the GUI, the second lens 

5 emphasizing a portion of the second area of the GUI, the portion of the second area emphasized 

6 by the second lens comprising a representation of multimedia information occurring between a 

7 third time (13) and a fourth time (ty), where (ti < 13 < 14 < 12); and 



Page 38 of 52 



Appl. No. 10/081,129 PATENT 

Amdt. dated February 11, 2008 

Reply to Office Action of April 20, 2007 

8 a code module for displaying, in a third area of the GUI, the representation of 

9 multimedia information occurring between t3 and 14, the representation of multimedia 

10 information displayed in the third area comprising a representation of information of the first 

1 1 type occurring between t3 and t/\ and a representation of information of the second type 

1 2 occurring between t3 and 14 . 



1 95. (Original) The system of claim 94 wherein the plurality of code modules 

2 further comprises: 

3 a code module for changing the position of the first lens in response to user input 

4 such that the first lens emphasizes a portion of the first area of the GUI comprising a 

5 representation of multimedia information occurring between a fifth time (t5) and a sixth time 

6 (t 6 ), where (t s < t 5 < t 6 < t e ), (t 5 * ti), and (t 6 * t 2 ); and 

7 responsive to the change in the position of the first lens, a code module for 

8 automatically displaying, in the second area of the GUI, the representation of multimedia 

9 information occurring between t5 and tg, the representation of multimedia information displayed 

10 in the second area comprising a representation of information of the first type occurring between 

11 15 and tg and a representation of information of the second type occurring between t5 and tg. 

1 96. (Original) The system of claim 94 wherein the plurality of code modules 

2 further comprises: 

3 a code module for changing the position of the second lens in response to user 

4 input such that the second lens emphasizes a portion of the second area of the GUI comprising a 

5 representation of multimedia information occurring between a fifth time (t5) and a sixth time 

6 (tg), where (tj < 15 < tg < t 2 ), (t5 * 13), and (tg * 14); and 

7 responsive to the change in the position of the second lens, a code module for 

8 automatically displaying, in the third area of the GUI, the representation of multimedia 

9 information occurring between t5 and tg, the representation of multimedia information displayed 
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1 0 in the third area comprising a representation of information of the first type occurring between t5 

1 1 and tg and a representation of information of the second type occurring between t5 and tg. 

1 97. (Original) The system of claim 94 wherein the plurality of code modules 

2 further comprises: 

3 a code module for displaying a third lens emphasizing a portion of the first area of 

4 the GUI comprising a representation of multimedia information occurring between t3 and 14 

1 98. (Original) The system of claim 97 wherein the plurality of code modules 

2 further comprises: 

3 a code module for changing the position of the second lens in response to user 

4 input such that the second lens emphasizes a portion of the second area of the GUI comprising a 

5 representation of multimedia information occurring between a fifth time (t5) and a sixth time 

6 (tg), where (tj < 15 < % < 12), £5 * ty), and (t£ * 14); and 

7 responsive to the change in the position of the second lens, a code module for 

8 automatically changing the position of the third lens such that the third lens emphasizes a portion 

9 of the first area of the GUI comprising a representation of multimedia information occurring 
10 between 15 and tg 

1 99. (Original) The system of claim 93 wherein: 

2 the information of the first type is video information; 

3 the information of the second type is audio information; 

4 the representation of the information of the first type comprises one or more video 

5 keyframes extracted from the video information; and 

6 the representation of information of the second type comprises text information 

7 obtained from transcribing the audio information. 

1 100. (Original) The system of claim 93 wherein: 

2 the information of the first type is video information; 
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3 the information of the second type is closed-caption (CC) text information; 

4 the representation of the information of the first type comprises one or more video 

5 keyframes extracted from the video information; and 

6 the representation of the information of the second type comprises text 

7 information included in the CC text information. 

1 101 . (Original) The system of claim 93 wherein the plurality of code modules 

2 further comprises: 

3 a code module for receiving information indicating a first topic; and 

4 a code module for analyzing the multimedia information stored in the multimedia 

5 document to identify one or more locations in the multimedia information that are relevant to the 

6 first topic; 

7 wherein the code module for displaying the representation of the multimedia 

8 information stored by the multimedia document occurring between t s and t e in the first area of 

9 the GUI comprises a code module for highlighting the one or more locations in the multimedia 

10 information displayed in the first area of the GUI; and 

1 1 wherein the code module for displaying the representation of multimedia 

12 information occurring between t^ and t2 in the second area of the GUI comprises a code module 

13 for highlighting the one or more locations in the multimedia information that occur between 

14 times t\ and t2- 

1 102. (Original) The system of claim 93 wherein the plurality of code modules 

2 further comprises: 

3 a code module for receiving input indicating selection of a portion of the 

4 multimedia information occurring between a selection start time and a selection end time; and 

5 a code module for performing a first operation on the portion of the multimedia 

6 information occurring between the selection start time and the selection end time. 
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1 103. (Currently amended) A system of displaying multimedia information 

2 stored in a multimedia document on a display, the multimedia information comprising video 

3 information and information of a first type, the system comprising: 

4 a display; 

5 a processor; and 

6 a memory coupled to the processor, the memory configured to store a computer 

7 program; 

8 wherein the processor is operative with the computer program to: 

9 display a graphical user interface (GUI) on the display; 

10 display, in a first section of a first area of the GUI, a first set of one or 

1 1 more video keyframes extracted from the video information occurring between a start time (t s ) 

12 and an end time (t e ) associated with the multimedia document, where (t e > t s ); 

13 display, in a second section of the first area of the GUI, text information 

14 corresponding to the information of the first type occurring between t s and t e ; 

15 display a first lens moveable in response to user input over representations 



16 of multimedia information displayed in the GUI, the first lens emphasizing a portion of the first 

17 section of the first area occurring between a first time (ti ) and a second time O2) and a portion of 

1 8 the second section of the first area occurring between t\ and t2, the emphasized portion of the 

19 first section of the first area comprising a second set of one or more video keyframes extracted 

20 from the video information occurring between t\ and t2, the emphasized portion of the second 



21 section of the first area comprising text information corresponding to information of the first type 

22 occurring between t^ and t2, wherein the second set of one or more keyframes is a subset of the 

23 first set of one or more keyframes and (t s < t \ < t2 < t e ); 

24 display the second set of one or more keyframes in a first section of a 

25 second area of the GUI; and 

26 display text information corresponding to the information of the first type 

27 occurring between t \ and t2 in a second section of the second area of the GUI. 
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1 104. (Currently amended) The system of claim 103 wherein the processor is 

2 operative with the computer program to: 

3 display a second lens moveable in response to user input over representations of 

4 multimedia information displayed in the GUI, the second lens emphasizing a portion of the first 

5 section of the second area and a portion of the second section of the second area, the emphasized 

6 portion of the first section of the second area comprising a third set of one or more video 

7 keyframes extracted from the video information occurring between a third time (t3) and a fourth 

8 time (ty), the emphasized portion of the second section of the second area comprising text 

9 information corresponding to information of the first type occurring between t3 and ty, wherein 

10 the third set of one or more video keyframes is a subset of the second set of one or more video 

1 1 keyframes and (t\ < 13 < 14 < t2); 



12 display a keyframe from the third set of one or more keyframes in a first section 

13 of a third area of the GUI; and 

14 display text information corresponding to the information of the first type 

15 occurring between t3 and 14 in a second section of the third area of the GUI. 

1 105. (Original) The system of claim 103 wherein the processor is operative 

2 with the computer program to: 

3 display a second lens emphasizing a portion of the first section of the second area 



4 and a portion of the second section of the second area, the emphasized portion of the first section 

5 of the second area comprising a third set of one or more video keyframes extracted from the 

6 video information occurring between a third time O3) and a fourth time O4), the emphasized 

7 portion of the second section of the second area comprising text information corresponding to 

8 information of the first type occurring between t3 and t4, wherein the third set of one or more 

9 video keyframes is a subset of the second set of one or more video keyframes and (t^ < 13 < 14 < 
10 t 2 ); 
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1 1 output video information starting from 13 or from 14 or from a time between 13 

12 and 14 in a first section of a third area of the GUI; and 

13 display text information corresponding to the information of the first type 

14 occurring between t3 and 14 in a second section of the third area of the GUI. 

1 106. (Original) The system of claim 103 wherein the information of the first 

2 type is audio information, and the text information corresponding to the information of the first 

3 type is obtained from transcribing the audio information. 

1 107. (Original) The system of claim 103 wherein the information of the first 

2 type is closed-caption (CC) text information, and the text information corresponding to the 

3 information of the first type is extracted from the CC text information. 

1 108. (Currently amended) The system of claim 103 wherein the multimedia 

2 information stored by the multimedia document further comprises slides information, and 

3 wherein the processor is operative with the computer program to: 

4 display, in a third section of the first area of the GUI, a first set of one or more 

5 slides extracted from the slides information occurring between t s and t e , wherein the first lens 

6 emphasizes a portion of the third section of the first area comprising a second set of one or more 

7 slides extracted from the slides information occurring between ti and t^, the second set of one or 

8 more slides is a subset of the first set of one or more slides; and 

9 display the second set of one or more slides in a third section of the second area of 
10 the GUL 

1 109. (Original) The system of claim 1 08 wherein the processor is operative 

2 with the computer program to: 

3 display a second lens emphasizing a portion of the first section of the second area, 

4 a portion of the second section of the second area, and a portion of the third section of the second 

5 area, the emphasized portion of the first section of the second area comprising a third set of one 

6 or more video keyframes extracted from the video information occurring between a third time 
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7 (13) and a fourth time (14), the emphasized portion of the second section of the second area 

8 comprising text information corresponding to information of the first type occurring between t3 

9 and ty, the emphasized portion of the third section of the second area comprising a third set of 

10 one or more slides extracted from the slides information occurring between t3 and 14, wherein 

1 1 the third set of one or more video keyframes is a subset of the second set of one or more video 

12 keyframes, the third set of one or more slides is a subset of the second set of one or more slides, 

13 and (ti < t3 < t4 < t2); 



14 display at least one keyframe from the third set of one or more video keyframes in 

15 a first section of a third area of the GUI; 

16 display text information corresponding to the information of the first type 

17 occurring between t3 and 14 in a second section of the third area of the GUI; and 

1 8 display at least one slide from the third set of one or more slides in a third section 

19 of the third area of the GUI . 

1 110. (Original) The system of claim 103 wherein the multimedia information 

2 stored by the multimedia document further comprises whiteboard images information, and 

3 wherein the processor is operative with the computer program to: 

4 display, in a third section of the first area of the GUI, a first set of one or more 



5 whiteboard images extracted from the whiteboard images information occurring between t s and 

6 t e , wherein the first lens emphasizes a portion of the third section of the first area comprising a 

7 second set of one or more whiteboard images extracted from the whiteboard images information 

8 occurring between t\ and t2, the second set of one or more whiteboard images is a subset of the 

9 first set of one or more whiteboard images; and 



10 display the second set of one or more whiteboard images in a third section of the 

1 1 second area of the GUI. 

1 111. (Original) The system of claim 110 wherein the processor is operative 

2 with the computer program to : 
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3 display a second lens emphasizing a portion of the first section of the second area, 

4 a portion of the second section of the second area, and a portion of the third section of the second 

5 area, the emphasized portion of the first section of the second area comprising a third set of one 

6 or more video keyframes extracted from the video information occurring between a third time 

7 (t3) and a fourth time (14), the emphasized portion of the second section of the second area 

8 comprising text information corresponding to information of the first type occurring between t3 

9 and 14, the emphasized portion of the third section of the second area comprising a third set of 

10 one or more whiteboard images extracted from the whiteboard images information occurring 

1 1 between t3 and t4, wherein the third set of one or more video keyframes is a subset of the second 

12 set of one or more video keyframes, the third set of one or more whiteboard images is a subset of 

13 the second set of one or more whiteboard images, and (t\ < < H ^ ^ 

14 display at least one keyframe from the third set of one or more video keyframes in 

15 a first section of a third area of the GUI; 

16 display text information corresponding to the information of the first type 

17 occurring between t3 and 14 in a second section of the third area of the GUI; and 

1 8 display a whiteboard images from the third set of one or more whiteboard images 

19 in a third section of the third area of the GUI. 
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